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ABSTRACT 


This thesis develops a quantitative process for determining if the changes being made to 
the combat/weapon systems during a maintenance availability are significant and the risk 
those changes pose to the readiness of the ship. The purpose of this process is to assist 
decision-makers in determining if a ship should conduct Combat Systems Ship 
Qualification Trials (CSSQT) post-availability. The process uses warfare area criticality 
and type of change to determine the significance of the change. Risk is assessed by 
determining the likelihood the changes will impact maintenance readiness, operational 
readiness and safety. The final result of the process is a matrix that uses the significance 
and risk likelihood values to graphically represent the overall impact the availability has 
on the ship. Where a ship is plotted on the matrix will determine the recommendation for 
that ship. 
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EXECUTIVE SUMMARY 


This thesis develops a process for quantifying the impact a major overhaul/modernization 
has on a ship’s overall readiness in terms of the significance of changes and the 
likelihood of risk. The need for such a process is because there is not a quantitative 
process in place for designating a ship for Combat Systems Ship Qualification Trials 
(CSSQT). The Naval Surface Warfare Center Port Hueneme Division (NSWC PHD) 
created a process and this thesis investigates that process and develops a methodology to 
determine the impact to ship’s readiness that builds from the current work. The result is a 
tool for determining the significance of the maintenance availability, a tool for assessing 
the likelihood of risk, and a process for presenting the results in a manner that allows 
decision-makers to easily determine if a ship should be designated to conduct a CSSQT 
based on the overall impact to ship’s readiness. 

The governing instruction for CSSQT, NAVSEAINST 9093.1C, states that a ship 
will be designated for a CSSQT if it is undergoing a significant overhaul/modemization 
availability. The instruction does not describe what constitutes significant; therefore, the 
first step is to make a decision about designating a ship for CSSQT is to determine if the 
availability is significant. NSWC PHD designed a procedure for determining significance 
and called it the CSSQT Decision Tool. A functional decomposition was used to review 
the tool and the following problems were discovered: 

• The name of the tool does not accurately reflect the purpose of the tool. 
This is a minor issue but it is important that the name of the tool describes 
the intended purpose. The current tool is called the CSSQT Decision Tool 
which indicates the purpose of the tool is to make a decision. The review 
determined that the actual purpose of the tool is to assess the significance 
of changes being made to the combat/weapon systems. To better reflect 
the purpose of the tool the name has been changed to Significance 
Assessment tool 

• There is an inaccurate accountability of all combat/weapon systems. Ten 
ships were assessed using NSWC PHD’s process and the review of the 
assessments found a majority of the ships had an inaccurate accountability 
of the combat/weapon systems onboard the ship. The assessments were 
missing systems, had the wrong systems listed and grouped training 
systems together. The training systems were grouped because they were 
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not critical to any warfare area and therefore any changes made to those 
systems did not need to be accounted for. Training systems are a major 
part of the crew’s readiness and need to be assessed individually in the 
same manner as the other systems. To achieve an accurate assessment, 
careful attention to include the correct systems must occur. 

• The significance calculation result is ambiguous. The result of the process 
is a percentage, but it is unclear what the value is a percentage of. The best 
approach for showing significance is to use an interval scale where a 
significance of 10 is extremely significant and a significance of 0 is not 
significant. 

• The significance calculation does not account for non-critical systems. 
Systems not critical to a warfare area are not accounted for in the 
assessment, even if the system is being altered. CSSQT validates and 
verifies all combat/weapon systems, so it is important every system is 
included in the assessment. A weight is given to non-critical systems and 
added to the function to determine significance. 

• A completed assessment can result in a spreadsheet that contains empty 
cells and cells that contain FALSE as an entry. Systems not being changed 
are left blank on the assessment spreadsheet which leads to invalid 
calculations and cells that contain FALSE. This practice makes the 
assessment look incomplete. To solve this, a category was added to 
indicate that a system is not being changed during the availability 

The likelihood of risk to a ship’s readiness can be used as an indicator of how the 
maintenance availability impacts a ship. This thesis developed a method for determining 
the likelihood a ship’s readiness will be impacted by the changes being made to the 
combat/weapon systems. The risk assessment tool developed by NSWC PHD was 
compared to the new methodology and the following issues were discovered: 

• The risk areas and risk factors do not align with the objectives outlined in 
NAVSEAINST 9093.1C. The risk areas proposed by NSWC PHD are 
Effectiveness, Readiness, Safety and Personnel but the instruction 
specifically states that the objectives of CSSQT are: “To ensure a ship’s 
combat system has a high level of operational readiness, maintenance 
readiness, and is safe” (NAVSEA 2006). The instruction also outlines the 
factors that are assessed to ensure the objectives are met. Aligning the risk 
assessment with the instruction ensures the assessment fully captures the 
intent of CSSQT. 

• A large number of risk factors required subjective input from outside 
sources. The risk assessment is intended to be quantitative, not qualitative, 
and the proposed process uses subjective assessments to determine risk. 
The assessment must be consistent for each ship being evaluated to ensure 
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the results are consistent. Relying on subjective assessment can lead to 
inconsistencies and can also make completing the assessment difficult. All 
subjectivity has been removed from the risk assessment and it relies only 
on data that can be found in on-line databases. 

• The output of the assessment is ambiguous. The result of the assessment is 
in the form of a percentage, but it is unclear what this percentage 
represents. The same scale that is used in the Significance Assessment will 
be used for the risk assessment, where 10 is extremely likely and 0 is not 
likely. 

The final result is a process that uses the significance assessment and risk 
assessment to determine the overall impact to a ship’s readiness. To accomplish this, a 
central spreadsheet for the results of the assessments was created. This is called the 
CSSQT Determination Dashboard. It collects the results from the assessments and 
determines a score for the significance and risk likelihood. These scores are plotted on 
the Determination Matrix, shown in Figure 1, for each ship being assessed. The plot 
provides a graphical view of the results and the quadrants of the matrix are labeled to 
give a recommendation based on the significance and likelihood. 
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I. 


INTRODUCTION 


This thesis will investigate and determine a quantitative procedure to assess the 
overall impact to a ship’s operational readiness after a major modernization. The 
procedure is required as part of a new systematic process for designating a ship to 
conduct Combat System Ship Qualification Trials (CSSQT). Naval Sea Systems 
Command, Port Hueneme Division (NSWC PHD) determined that there is a ""Lack of an 
objective and validated process to determine CSSQTsf there is an ""Inability to predict 
risk for post availability impacts 3 to 4 years in advance,” and there are ""Multiple 
interpretations of ‘significant’” (B. Hazel and S. Matthews, unpublished Powerpoint 
slide). In 2012, NSWC PHD introduced a new process for determining which ships 
should be designated for CSSQT and the process includes a tool for assessing the 
significance and a tool for assessing the risk. This thesis will present a methodology to 
review the two tools and present a new procedure for assessing overall impact to a ship’s 
readiness. 

A. PROBLEM STATEMENT 

According to NAVSEAINST 9093.1C, “CSSQTs will be scheduled for all surface 
combatants, aircraft carriers, and amphibious ships completing new construction or 
significant conversion/modernization availabilities” (2006). The problem is how to define 
a significant conversion/modemization. Prior to 2012, it was the responsibility of the 
“applicable Program Executive Offices (PEOs)/Program Managers (PMs) in coordination 
with the Type Commanders, CSSQT Certification Authority and Naval Sea Systems 
Command (SEA 62) to designate a ship for CSSQT post-availability” (NAVSEA 2006) 
based on subjective assessment and the availability of funding. NSWC PHD (CSSQT 
Certification Authority) has determined the need for a process that removes subjectivity 
and relies on a quantitative assessment to determine the significance of combat/weapon 
system changes made during a maintenance availability in order to make a decision if a 
CSSQT is warranted for a specific ship. Eurthermore, NSWC PHD deemed it necessary 
to assess the risk associated with the significance of the conversion/modemization. An 
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assessment of the significance and the risk to the ship will give decision-makers the 
information to fully understand the nature of the modernization and make a better 
decision when designating a ship for CSSQT. A procedure that can be applied to all ships 
scheduled for a conversion/modernization needs to be designed that will assess the 
significance and risk the conversion/modemization will have on the ship’s readiness. 

B. SCOPE 

CSSQT is performed on every class of surface combatant ships and it is necessary 
to assess each ship undergoing a conversion/modernization, but this research will restrict 
the assessment to amphibious ships and aircraft carriers. The research will analyze ships 
conducting a maintenance availability in fiscal year 2015. The focus will be on reviewing 
the tools created by NSWC PHD and developing an objective process for determining the 
significance and risk associated with a major conversion/modernization. 

C. ORGANIZATION 

The next chapter will discuss the background of CSSQT and present the current 
work being done by NSWC PHD. The third chapter describes the methodology used to 
review the current work and the process used to develop the new risk and significance 
tools. The fourth chapter will present the results of the methodology. The final chapter 
will discuss conclusions and recommendations for further research. 
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II. BACKGROUND 


A. COMBAT SYSTEMS SHIP QUALIFICATION TRIALS 

CSSQT is a major evolution that requires years of planning and two to three 
months to execute depending on the complexity of the combat systems on the individual 
ship. The execution of a CSSQT is governed by NAVSEA Instruction 9093.1C, Combat 
System Ship Qualification Trials for Surface Ships. The instruction outlines the purpose, 
policy, objectives, definitions, guidelines and the planning process for any surface ship 
designated to conduct a CSSQT. Purpose and policy are important to understanding why 
CSSQTs are conducted and what is required to designate a ship to perform one. They also 
provide valuable information on how to approach creating a process for making a 
recommendation a ship conduct a CSSQT. The purpose of CSSQT is: 

To verify and validate that an individual ship’s combat/weapon systems 
have been installed correctly and can be operated and maintained in a safe 
and effective manner. This is accomplished by assisting ship’s force in 
achieving (1) a sustainable level of combat/weapon system operational 
readiness and (2) a maintainable level of material readiness. (NAVSEA 
2006) 

The purpose indicates after a successful completion of CSSQT the ship will be 
able to operate and maintain the combat/weapon systems at a higher level than if it had 
not conducted a CSSQT. It can be inferred a ship should conduct a CSSQT if there are 
circumstance that reduce the ship’s ability to operate and maintain the combat/weapon 
systems. The policy of CSSQT provides the circumstances and gives guidance to when a 
ship should conduct a CSSQT. 

CSSQTs will be scheduled for all surface combatants, aircraft carriers, and 
amphibious ships completing new construction or significant 
conversion/modernization availabilities. Designation of post availability 
ships for CSSQTs will be made by applicable Program Executive Offices 
(PEOs)/Program Managers (PMs) in coordination with the Type 
Commanders, CSSQT Certification Authority and Naval Sea Systems 
Command (SEA 62). These designations will be based on the extent of 
combat system changes made during the availability or the cumulative 
effect of combat/weapon systems changes since the last CSSQT. 
Determinations to conduct CSSQTs and their scope will be developed by a 
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collaborative system engineering process. Following CSSQT, each 

applicable mission area will be qualified to be safe and effective based on 

combat/weapon system performance and observed crew proficiency. 

(NAVSEA 2006) 

A ship is designated to execute a CSSQT at an annual review board conducted by 
SEA 62. The board looks at all surface ships conducting a maintenance availability in the 
next four years and determines if the ship requires a CSSQT. Recommendations from 
PEOs, PMs, TYCOMs and CSSQT Execution Agency, type of availability and the 
number of years since the ship conducted its last CSSQT allow the review board to make 
a decision if a ship should conduct a CSSQT. Once a ship is designated to conduct the 
trials, a test plan will be developed based on combat system requirements. The test plan is 
ship specific in which the scope is unique, but the objectives are the same. 

The scope of a CSSQT is based on the warfare areas of the ship. The warfare 
areas that can be included in a CSSQT are: 

• Air Defense Warfare (ADW) 

• Strike Warfare (STW) 

• Surface Warfare (SUW) 

• Undersea Warfare (USW) 

• Expeditionary Warfare (EXW) 

• Electronic Warfare (EW) 

CSSQT assess all combat/weapon systems that support each warfare area. A 
combat/weapon system is defined as a functional grouping of all the shipboard 
equipment, computer programs, personnel and documentation, plus other critical systems 
that are designed to detect, track, identify, communicate, process, evaluate, and execute 
the engagement of enemy forces, either actively passively (NAVSEA 2006). Not all ships 
are capable of all the above listed warfare areas so the scope of each CSSQT will depend 
on the capabilities of the ship. Regardless of the scope of the CSSQT, the objectives 
remain constant. 

The objectives of CSSQT are “to verify and validate that the combat/weapon 
systems have been installed correctly and can be operated and maintained in a safe and 

effective manner” (NAVSEA 2006). Verification and validation is accomplished by 
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subject matter experts (SMEs) assisting the crew with Planned Maintenance System 
(PMS) actions and also conducting in-port and at-sea combat system exercises. The 
exercises use live and/or simulated environments to stress the combat systems and ship’s 
force in order to determine the level of maintenance readiness and operational readiness. 
Overall readiness is assessed using Measures of Effectiveness (MOEs) for adequacy of 
system documentation, logistics support, maintenance, and operational readiness. 
Assessing the MOEs and meeting the objectives of the CSSQT ensure the ship has a safe 
and effective combat system and is prepared for future training and operations. 

CSSQT is a valuable experience for the ship’s force because of the opportunity to 
work closely with subject matter experts (SMEs) while the crew learns to maintain the 
equipment and experiences how to operate it in real-world situations. The other benefit of 
CSSQT is the opportunity for engineers to conduct integrated Developmental and 
Operational Testing (DT/OT), to perform tactics validation and also conduct fleet 
training. The problem is a ship may only conduct one or two CSSQTs during the entire 
life of the ship, but combat and weapons systems are continuously being upgraded and/or 
added through the Navy Modernization Plan (NMP). It would be ideal for a ship to 
conduct a CSSQT after every major modernization, but due to the cost to conduct one it 
is not feasible. Based on the assessment of risk and significance it would be possible to 
recommend a CSSQT that is scoped down to ensure the major changes to the 
combat/weapon systems are validated and verified. 

B. CURRENT WORK 

NSWC PHD created the CSSQT Determination Process in 2012 to capture the 
process outlined in NAVSEAINST 9093.1C. It introduces two tools for determining the 
significance of changes being made to the ship and assessing the risk associated with the 
changes. Eigure 2 shows the beginning of the determination process. The full flow chart 
can be found in Appendix A. 
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The process begins prior to the System Engineering Board (SEB) which occurs 
60 days before the CSSQT Annual Review Board (ARB) (not pictured in Eigure 2). The 
SEB reviews each ship that is scheduled to undergo an overhaul/modemization 
availability in the next four years. The SEB uses the assessments from the CSSQT 
Decision Tool process and the CSSQT Risk Assessment process to make a 
recommendation to the sponsors/CSSQT community that a ship should or should not 
conduct a CSSQT. Ships reviewed in previous years are reassessed for any changes to the 
ship’s availability. The Decision Tool and Risk Assessment processes allow the SEB to 
make a more informed decision about designating ships for CSSQT. Each process will be 
discussed in more detail in the following sections. 

1. CSSQT Decision Tool Process 

The purpose of the CSSQT Decision Tool process is to determine the significance 
of the overhaul/modernization availability. The decision tool is a simple spreadsheet 
which lists the combat/weapon systems onboard the ship and assigns weights to each 
system based on the criticality to each warfare area and the type of modernization. 
Weights used for criticality and type of modernization were determined by the developers 
of the process. A detailed analysis of how the weights affect the results will be discussed 
in Chapter III. The criticality weight is multiplied by the modernization weight to yield a 
warfare area qualification weight. The sum of the warfare area qualification weights 
divided by the sum of the warfare criticality weights produces a percentage that 
represents the significance of changes being made to each warfare area. The average of 
the significance values represents the overall significance of the availability. The 
spreadsheet is populated using Warfare Interface System Engineering (WISE) database 
and Ship Change Documents (SCDs). 

NSWC PHD uses the WISE database to determine when ships will be entering 
into a modernization period and to determine which systems will be changed. The 
database contains the current combat and weapon systems onboard the ship, which 
software version the systems are running and also when scheduled upgrades to the 
systems will occur. Each availability has a corresponding Warfare System Interface 
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Diagram (WSID) that is assessable from the WISE database. The WSID is a graphic that 
shows all combat and weapon systems, how they are connected to the command and 
control system and which systems are being modified during the availability. It also 
contains a list of the applicable SCDs and a description of the changes. An example of a 
WSID can be found in Appendix B. The WSID is used to populate the decision tool 
spreadsheet with the systems and get the applicable SCD numbers, but the SCDs are 
located in another database. 

A SCD is the authorized document for any ship change during maintenance 
availabilities. All SCDs are located on the Navy Data Environment Navy Modernization 
Entitled Process database. SCDs contain a lot of information but the information used by 
NSWC PHD is about the type of modification being made to the system. It indicates if 
the system is being added to the ship, a new system is replacing an old system, the 
modification is adding a new capability, improving capability, improving performance, or 
improving infrastructure. This information is used in the decision tool to determine the 
modernization weight values. The weights used for each type of modification are shown 
in Eigure 3. If a system is not being changed it receives a value of EAESE because there 
is not a category for a system not being altered. 


Type 

Weight 

Replace with New 

1 

Add Capability 

1 

Improve Capability 

1 

Improve Performance 

0.5 

Improve Infrastructures 

0.25 


Eigure 2. Modification Weights (Erom B. Hazel and S. Matthews, 

unpublished Excel spreadsheet) 

The SCD does not show how vital a system is to a specific warfare area. The 
assessment of criticality is based on knowledge of how each system functions to support 
a specific warfare area. Eor example, a three-dimensional air search radar may be critical 
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to anti-air warfare but unnecessary for the mission of surface warfare because surface 
warfare requires a surface search radar. NSWC PHD uses the categories and weights 
shown in Figure 4 to apply criticality to each system. 


Category 

Comments 

Weight 

Critical 

Without this system cannot perform the 
mission (Mission Failure) 

1 

Necessary 

Without this system mission can be 
performed but at a reduced success rate 
(Reduced Performance) 

0.5 

Like to have 

Without this system mission can be 
performed but with reduced awareness 
(Reduces Situational Awareness) 

0.25 

Unnecessary for Mission 

System does not affect mission (No Effect) 

0 


Figure 3. Mission Area Criticality Categories and Weights (From B. Hazel 
and S. Matthews, unpublished Excel spreadsheet) 

The decision tool allows users to select the criticality category and modernization 
type from a drop down list. When a selection is made the weight value is automatically 
filled in and the spreadsheet will update the significance percentage. If the value is 
greater than 25% then a CSSQT is recommended and indicated on the spreadsheet. A 
completed decision tool is in Appendix C. After the CSSQT Decision Tool process the 
next step is to conduct a risk assessment. 

2. Risk Assessment Tool Process 

The risk assessment tool process has not been put into practice like the Decision 
Tool. It is still under development but NSWC PHD has developed a potential tool that 
could be used to assess the risk of a ship undergoing a major modernization. The tool 
assesses the risk to effectiveness, personnel, safety and readiness through a combination 
of subjective evaluations of the availability by Subject Matter Experts (SMEs), historical 
data about the ship, and current data about the availability. The developers of the tool. 
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Bob Hazel and Sandra Matthew, from NSWC PHD defined the four areas of risk in the 
following way: 


a. Effectiveness 

The combat system is not able to perform its mission. Effectiveness risk is 
determined by the magnitude of changes impacting the combat/weapon system 
functionality (B. Hazel and S. Matthews, unpublished Excel spreadsheet). 

b. Personnel 

The combat systems cannot be operated effectively by the crew. Personnel 
risk is determined by the magnitude of changes impacting the crew’s ability to operate 
and maintain systems (B. Hazel and S. Matthews, unpublished Excel spreadsheet). 

c. Safety 

Personal injury or death and/or equipment damage. Safety risk is assessed 
by the issues associated with a ship’s modification, in conjunction with historical data, to 
determine the impact with individual or aggregate consequences (B. Hazel and S. 
Matthews, unpublished Excel spreadsheet). 

d. Readiness 

The combat system is not available/ready to perform the mission. 
Readiness risk is determined by the magnitude of changes impacting the combat/weapons 
systems ability to perform its mission (B. Hazel and S. Matthews, unpublished Excel 
spreadsheet). 

There are 18 risk factors assigned to one or more of the categories that are 
used to assess the risk and Eigure 5 shows how each risk factor is assigned to each risk 
category (an X indicates that risk factor is used in assessing the risk category). 
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Effective 

Personnel 

OJ 

Material Readiness 

RISK FACTORS 

D 



X 

Years since last CSSQT 

D 



X 

Years since last live firing 

1 X 

X 

X 

X 

Complexity Change of Combat System 

1 

X 


X 

Number of Changes in 1LS Support (Tech Doc) 

1 

X 


X 

Number of Changes in 1 LS Support (Test Equip) 


X 


X 

Number of Changes in 1 LS Support (Spares) 


X 


X 

Number of Changes in 1 LS Support (PMS) 

X 

X 


X 

Training Program Planned or Developed/Delivered (AIT) 

X 

X 


X 

Training Program Planned or Developed/Delivered (LETS) 

X 

X 


X 

Planned ATG training 

X 

X 

X 


Certifications/Qualifications 1 mpacted 

X 

X 



Number of open R1 Trouble Reports (TRs) 

X 

X 



Ship's force ability to perform the mission 


X 


X 

Known System 1 ncidents of risks 



X 


Known Safety Incidents of risks 


X 



Manpower Requirements adequately reflected 

X 

X 

X 


Project Officer/Project Engineer Evaluation of Risk 

X 

X 

X 


Baseline Manager/Equivalent (i.e. Test Con./Dir. Evaluation of Risk 


Figure 4. Risk Factor Assignments (From B. Hazel and S. Matthews, 

unpublished Excel spreadsheet) 


The risk factors and how they are assessed is shown in Figure 6. 
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RISK FACTORS Assessment 


Years since last CSSQT 

Enter number of years since last CSSQT 

Years since last live firing 

Enter number of years since lastfring (reduces) 

Complexity Change of Combat System 

Modernizaton Impact percentage taken from the Decision tool 

Number of Changes in 1LS Support (Tech Doc) 

PUBSAT: Calculate using SCO's. CountnumberofSCDs identified as impactng 
TECH DOC (Counting multiple SCDs to a single system only once)(X)/Total 
number of ships systems (XX) Tech docs checked/tolal number of systems 

Number of Changes in 1 LS Support (Test Equip) 

TESAT: Calculate using SCO's. CountnumberofSCDs idenied as impacting 
TEST EQUIP (Counting multple SCDs to a single system only once)(X)/Tolal 
number of ships systems (XX)ech docs checked/tolal number of systems 

Number of Changes in 1 LS Support (Spares) 

LOGSAT: Calculate using SCO's. CountnumberofSCDs idenffed as impactng 
SPARES (Counfng multple SCDs to a single system only once)(X)/Total number 
of ships systems (XX) ech docs checked/total number of systems 

Number of Changes in 1 LS Support (PMS) 

PUBSAT: Calculate using SCO's. CountnumberofSCDs identted as impactng 
PMS (Counfng multple SCDs to a single system only once)(X)/Total number of 
ships systems (XX) ech docs checked/tolal number of systems 

Training Program Planned or Developed/Delivered (AIT) 

Modernizaton Rep/SME 

Training Program Planned or Developed/Delivered (LBTS) 

B/L Mgr/SME 

Planned ATG training 

Training SME 

Certifications/Qualifications 1 mpacted 

SCDSecfon 15/23 

Number of open R1 Trouble Reports (TRs) 

NWSCP R1 defniton 

Ship's force ability to perform the mission 

NWSCP-Aggregate Assessment of numberofTactcs, Techniques, and 
Procedures (TTP), workarounds, limitafons, and resfricfons indicates ship's force 
can perform fhe mission PRI 3/4's) 

Known System Incidents of risks 

Modernizaton Division System Engineer's evaluate known system incidents of 
risks 

Known Safety 1 ncidents of risks 

SME evaluate known safety Incidenfs of risks 

Manpower Requirements adequately reflected 

Fit/Fill 

Project Officer/Project Engineer Evaluation of Risk 

Evaluaton of Ship Risk infroduced from fhe specitc changes occurring/occurred 
to a specitc ship at a specitc Ship yard wifi a specitc Training Team 

Baseline Manager/Equivalent (i.e. Test Con./Dir. Evaluation of Risk 

Evaluaton of Baseline Risks Infroduced from fie specitc changes 
occurring/occurred to a specitc ship 


Figure 5. Risk Factors and How they are Assessed (From B. Hazel and 
S. Matthews, unpublished Excel spreadsheet) 

To complete the risk assessment each risk factor is assigned a weight 
based on the assessments that were conducted. Figures 7 through 10 show criteria and 
weights for each risk factor. 
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Figure 6. Effectiveness Criteria and Weights (From B. Hazel and 
S. Matthews, unpublished Excel spreadsheet) 



Figure 7. Personnel Criteria and Weights (From B. Hazel and 
S. Matthews, unpublished Excel spreadsheet) 
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Risk Factors 

High 1.0 

Medium .5 

Low .25 

Not Effected 0 

Com plexity Change of Com bat 


49%-25% 

24%-1% 

0% 

Certifications/Qualifications 


3-2 

1 

0 

Known Safety Incidents of risks 


Medium 

Low 

None 

Project Officer/Project Engineer 


Medium 

Low 

None 

Baseline Manager/Equivalent (i.e. 

Test Con./Dir. Evaluation of Risk 


Medium 

Low 

None 


Figure 8. Safety Criteria and Weights (From B. Hazel and S. Matthews, 

unpublished Excel spreadsheet) 



Figure 9. Readiness Criteria and Weights (From B. Hazel and S. Matthews, 

unpublished Excel spreadsheet) 


After the assessment is completed the results are entered into a 
spreadsheet that calculates the risk score for each risk category and the overall risk score 
for the ship. An example of a completed risk assessment is shown in Figure 11. 
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ENTER SHIP NAME AND FY OF CSSQT 

Risk Assessment 

Risk Factors 

ENTER DATA 

Effective 

Personnel 

Safe 

Material 

Readiness 

Years since last CSSQT 

2 

0 



0 

Years since last live firing 





0.0 

Com plexity Change of Com bat System 


.5 

.5 

.5 

.5 

Number of Changes in ILS Support (Tech Doc) 

19% 


.25 


.25 

Num ber of Changes in ILS Support (Test Equip) 

52% 


1.0 


1.0 

Number of Changes in ILS Support (Spares) 

4% 


.25 


.25 

Number of Changes in ILS Support (PMS) 

4% 


.25 


.25 

Training Program Planned or Developed/Delivered 
(AIT) 

Not Planned 

1.0 

1.0 


1.0 

Training Program Planned or Developed/Delivered 
(LBTS) 

Planned 

.25 

.25 


.25 

Planned ATG training 

Planned 

.25 

.25 


.25 

Certifications/Qualifications Impacted 

13 

.5 

.5 

.5 


Number of open R1 Trouble Reports (TRs) 

5 

1.0 

1.0 



Ship’s force ability to perform the mission 

15 

1.0 

1.0 



Known System Incidents of risks 

Medium 


.5 


.5 

Known Safety Incidents of risks 

Medium 



.5 


Manpower Requirements adequately reflected 

Yes 


.25 



Project Qfficer/Project Engineer Evaluation of Risk 

Low 

.25 

.25 

.25 


Baseline Manager/Equivalent (i.e. TestCon./Dir. 
Evaluation of Risk 

Low 

.25 

.25 

.25 


TOTAL 

44.64% 

45.45% 

50.00% 

40.00% 

38.64% 


Figure 10. Completed Risk Assessment Example (From B. Hazel and 
S. Matthews, unpublished Excel spreadsheet) 


The risk score for each category is the sum of risk factor weights divided 
by the total number of risk factors for the specific category. The total score for the overall 
risk assessment is the sum of all risk factor weights divided by the total number of risk 
factors. Each risk category is given a severity rating based on the category score: 

• >51% Red Severity Rating 

• >26% and <51% Yellow Severity Rating 

• >1% and <26% Green Severity Rating 

• 0% and <1% Blue Severity Rating 

The higher the score the greater risk the ship may experience post¬ 
availability to effectiveness, personnel, safety and readiness. The risk assessment gives 
the CSSQT Certification Authority and other decision-makers the ability to make a 

15 

























stronger recommendation for a ship to be designated to conduct a CSSQT. A low risk 
assessment score indicates the changes being made to the ship will not have an impact on 
the readiness of the ship and therefore a CSSQT may not be required even if the CSSQT 
decision tool recommends a CSSQT. 

The next chapter will investigate the current work to find if there is a 
better approach to the two tools. The tools are undeveloped and lack validation due to the 
short time they have been use. The goal is to develop a tool that captures both the 
significance and risk in one tool/process rather than having two separate tools that are not 
coupled. 
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III. METHODOLOGY 


The approach used for this research consists of three stages. The first stage 
develops a methodology for reviewing the CSSQT Decision Tool process and determines 
if it is achieving the desired results of the process. The second stage creates a 
methodology for assessing risk and reviews the Risk Assessment process and. The third 
stage develops an approach to present significance and risk in a manner that gives 
decision-makers a better understanding of how the modernizations are impacting the 
ships in a given fiscal year. This chapter will only present the methodologies for each 
stage. The results of the methodologies will be presented in Chapter IV. 

A. CSSQT DECISION TOOL REVIEW 

The goal of this stage is to create a methodology to review the current CSSQT 
Decision Tool process being used by NSWC PHD. The methodology needs to determine 
what the process is trying to accomplish and what is needed to do this. Another way to 
say this would be what is the function of the process and what functions are required to 
achieve the overall function. When looked at in this way the best approach is to use a 
functional decomposition. 

A functional decomposition will show what functions the tool will need to do to 
achieve the overall purpose. The first step is to define the overall function of the process. 
The second step is to create a functional decomposition and the third step is to examine 
the current tool to see if it performs the functions. The Decision Tool review will use a 
completed assessment of 10 ships conducting modernization availabilities in fiscal year 
2015. 

The purpose of the CSSQT Decision Tool process is to determine if the 
modifications being made to the ship during a maintenance availability are significant. 
The need for this determination is because NAVSEAINST 9093.1C states that ships will 
conduct a CSSQT if the overhaul/modernization is significant. Defining significant is 
essential to the assessment. Significant, for the purpose of this process, is defined as any 
change or combination of changes that reduce the overall combat readiness of a ship. 
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Using the definition, a maintenance availability is not significant if the changes being 
made to the combat/weapon systems do not reduce the ship’s ability to execute its 
assigned missions. However, there can be degrees of how significant the changes are 
beyond not significant. If the impact on the combat readiness is minimal then it can be 
determined that the overhaul is somewhat significant. When the changes greatly reduce 
the combat readiness than the overhaul is extremely significant. To determine just how 
significant the changes are, a scale needs to be implemented to quantify the changes 
being made to the ship. 

For the CSSQT Decision Tool to accurately determine how significant the 
overhaul is, it must first define a quantifiable scale for significance. Significance is the 
quantitative value for how significant the modifications the overhaul is; therefore, a finite 
interval scale can be used to assign a value to how significant the changes are. A scale 
that spans from 0 to 10, where 0 is not significant and 10 is extremely significant will 
allow the process to assign values for significance. The overall function of the CSSQT 
Decision Tool is to quantify the impact to each system in terms of significance so the 
aggregate of all the systems yields a value that can be used on an interval scale to 
determine if the overhaul is significant. 

The approach for determining the significance of changes requires quantifying the 
impact the changes have on the combat system. The functions of the process need to 
provide a quantitative analysis of the changes being made to the different systems based 
on their importance to the warfare areas and the type of modification being made to the 
systems. The result of this analysis will yield the level of change being made to the 
combat/weapon system and the level can be used on an interval scale to show if the 
change is significant. The process will have top-level functions and sub-functions that are 
required to ensure the overall function is possible. These functions are shown in the 
functional decomposition in Figure 12. The following sections describe each function and 
reviews of the CSSQT Decision Tool’s ability to perform the functions. 
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Figure 11. Significance Functional Decomposition 

1. Account for All Systems 

The first step in determining significance is to accurately account for all 
combat/weapon systems installed on the ship. Ensuring all systems are included in the 
assessment will reduce the risk of omitting a major change that could potentially impact 
the ship’s readiness. The current process does list the systems but the review found the 
list was inaccurate and also grouped systems based on common functions. The list of 
systems in the reviewed completed assessment contained systems that were not installed 
and also did not list installed systems. Training systems were grouped into one system 
which resulted in a lack of understanding which system is being modified, in the case 
there is a modification. In most cases, at least one of the three training systems onboard a 
ship were being modified in some way; but, it was unclear which system was being 
affected. Training systems are considered non-critical to the mission of the ship; 
however, they are vital to ensuring the crew is trained to execute the mission. They must 
be treated in the same manner as other major systems and each system evaluated 
individually versus grouped. 

The class of ship may be the same but the systems onboard the ship can differ 
immensely. It is vital to the process that each system is accounted for so changes are 
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assessed properly. The WISE database has the capability to allow users to download a list 
of systems, in spreadsheet format, onboard a ship post-availability. This capability must 
be used for each ship being assessed to ensure all systems are accounted for. The next 
function is to determine how critical each system is to each warfare area. 

2. Determine System Criticality 

System criticality is dependent on how the system integrates into the detect-to- 
engage path for each warfare area. The assessment is done by determining if the warfare 
area mission can still be accomplished without that specific system. The criticality will 
determine the weight that will be assigned to the system. The current process does apply 
mission area criticality to each system and the categories were discussed in Chapter II. 
The review found that the first three categories are appropriate for assessing the criticality 
but the Unnecessary for Mission category needs to be addressed. All systems onboard 
ships have some necessity in completing the mission. For this reason, the category needs 
to be changed to Non-Critical. This change makes it clear that a system may not directly 
be required for a mission area but is still important to the overall mission readiness of the 
ship. 


a. Apply Criticality Weights 

The purpose of a criticality weight is to quantify the criticality. A system 
is assigned a category based on how critical it is to the warfare area, but the category 
needs to be converted to a number for use in the formula for determining impact on the 
warfare area. The scale used for criticality is a non-dichotomous ordinal scale because it 
spans a spectrum of how critical a system is to a warfare area. Typically, a scale of this 
nature will have numbers associated with each category. For example, a survey of 
opinion might have the categories of completely agree, mostly agree, no opinion, mostly 
disagree, and completely disagree where the values for the categories would be 5, 4, 3, 2 
and 1, respectfully. The equal separation between values demonstrates a consistent 
decline if the magnitude of the categories. If the separation is not consistent in this type 
of scale than the concern is the full spectrum is not captured by the categories. 
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Capturing the full spectrum is important for determining the criticality 
weights. If a system is not critical, the next category should be indicative of the next level 
of criticality. The value should also represent one step down from the previous value. The 
range between the values can be considered the magnitude of difference between 
categories. When the separation between category values is equal, there is an equal 
magnitude of difference between categories indicating a consistent decrease from one 
category to the next. If the magnitude of difference is not equal, it could indicate another 
category is required to ensure the scale includes all required categories. The criticality 
categories. Critical, Necessary, Like to Have, and Non-Critical can be plotted on a 
number line, with equal intervals, that ranges from 0 to 1, and a system categorized as 
Critical would earn a value of 1 and a Non-Critical system would earn a value of 0. A 
Like to Have system would have a value of 0.33 and a Necessary system would have a 
value of 0.66. The equal separation ensures the categories represent the entire range of 
criticality. 

NSWC PHD gives a Critical system a weight of 1, a Necessary system a 
weight of 0.50 and a Like to Have system a weight of 0.25 indicating a Critical system is 
two magnitudes more critical as a Necessary system, but a Necessary systems is only a 
single magnitude more critical than a Like to Have system. The problem with this 
approach is there is no justification for why the separation between categories is different. 
It should be understood why a Critical system is more critical than a Necessary system, 
but NSWC PHD does not address the reason in the description of the tool. It is possible 
that the selection of weights does not affect the results of the analysis and therefore a 
justification of why the values used for the weights is not necessary. 

A sensitivity analysis on the values for the weights was conducted to 
determine if changing the values will result in a different outcome of the assessment. 
There are two possible outcomes of the assessment: CSSQT Required or No CSSQT 
Required. Out of the ten ships assessed, six resulted in No CSSQT Required with the 
weighting scheme used by NSWC PHD. The first analysis changed the weights so there 
was a single magnitude of difference between the categories and there was no change to 
the outcome of the assessment. The second analysis changed the value of the weight for 
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Necessary, Like to Have and Non-Critical systems to 0.75, 0.50, and 0.25, respectfully, 
and the results were not affected. Each analysis maintained the weight for a Critical 
system as 1 and the values for the other weights were separated by an equal magnitude. 
The analysis was performed to show if the values for the weight influence the outcome of 
the assessment, in both cases the outcome was not affected. 

The values for the criticality weights do not influence the results of the 
assessment and the methodology presented recommends changing the values. Changing 
the weight values corrects a problem with how the overall significance is calculated. The 
current process multiplies the criticality weight by the modification weight to determine 
the significance a change has on an individual system. When a non-critical system is 
being modified, it does not factor into the significance because a Non-Critical system has 
a weight of 0. To maintain an equal separation of weights, a Non-Critical system will 
have a weight of 0.25, a Like to Have system a weight of 0.50, and a Necessary system a 
weight of 0.75. The process for including systems that are considered Non-Critical will 
be discussed in the section describing the function of calculating noon-critical 
significance. 


3. Determine Type of Modification 

The type of modification evaluates if a systems is being altered during the 
availability and the nature of the alteration. There are five categories of modifications, the 
categories and the descriptions are shown in Figure 13. 


Type 

Comments 

Example | 

Replace with New 

This is a significant change to the combat 
system. Disrupts the DTE data path 

Replace ACDS with SSDS, SYS- 
2 with CEC 

Add Capability 

This is a significant change/addition to a 
mission area 

Add CIWS to a platform, 
NSSMS surface mode 

Improve Capability 

This is a significant change to a systems 
capability 

ESSM, IFF mode V 

Improve Performance 

This is a significant change to a systems 
performance 

NSSMS solid state 
transmitter, 48G 

Improve Infrastructures 

This is a significant change to a systems 
configuration 

SSDS open architecture 


Figure 12. Modification Types and Descriptions (From B. Hazel and 
S. Matthews, unpublished Excel spreadsheet) 

22 




The main issue with the type of modification categories is there is not a category 
for a system not being changed. This leads to the decision tool having blank cells that 
gives no information to the user. A blank cell could indicate that the system has not been 
evaluated or the type of change is unknown. A sixth category needs to be added to 
account for systems not being changed during the availability. 

a. Apply Modification Weights 

The purpose of applying weights to the modification type is to quantify the 
impact the modification has on the system. As discussed previously, the modification 
weight and the criticality weight are used to score the significance of the changes to a 
warfare area. The weighting scheme used by NSWC PHD give equal value to the first 
three types of modifications indicating the impact to the system is the same for each 
modification. The last two categories have a lesser amount of weight indicating the 
modification does not impact the system as much as the other modifications. The same 
scale used for the criticality weights is used for the modification weights. The first three 
modification types are assigned a weight of 1, the fourth type has a value of 0.50 and the 
last category is given a weight of 0.25. Again, there is an unequal separation between the 
types of modifications. The weighting scheme indicates a modification that replaces an 
old system or adds or improves the capability of a system will impact the system twice as 
much as a modification that improves the performance of the system and an improvement 
to performance has twice the impact as an improvement to infrastructure. Since the 
weights are assigned based on the potential impact the modification has on a system, it is 
possible the outcome of the assessment can be impacted by changing the values of the 
weights. 

Sensitivity analysis was performed to determine if the values of the 
weights are critical to the outcome of the assessment. Two analyses were conducted to 
determine if the recommendations of the Decision Tool can be changed if the values for 
the weights are altered. The first analysis increased the values of the Improve 
Performance and Improve Infrastructure to 0.90 and 0.50, respectfully. The second 
analysis decreased the weights to 0.30 and 0.10. The weights of the first three types of 
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modifications were not changed because it is understood the impact to the system is the 
greatest when one of these modifications occur. The outcome of the assessments of ten 
ships did not change in either of the two analyses. Further analysis would be required if 
the outcomes did change to determine the sensitivity of the model. This simple sensitivity 
analysis demonstrates the model is not sensitive to changing the weights and therefore, 
the weights will not be changed. 

4. Calculate Warfare Area Significance 

The purpose of this function is to determine the magnitude of impact the changes 
to the systems has on a warfare area. This is computed by multiplying the criticality 
weight by the modernization weight. If a system is not affected during the availability, 
the score will be zero because the weight of no change is zero. The highest score an 
individual system can receive is one. The sum of all the individual system scores will 
yield the impact to the warfare area. This value will range from 0 to 10 (in most cases, 
there is the possibility that the value could be greater than 10, but it is not likely) where 
10 indicates a significant amount of modification is being made to the combat/weapon 
systems. The review found the current process does this calculation, but does another 
calculation that is not necessary. 

NSWC PHD determines the significance of changes made to each warfare area by 
taking the sum of individual impact scores and dividing it by the sum of the criticality 
weights. The ratio of warfare area impact to warfare area criticality yields a value that is 
converted to a percentage. The percentage is what is used to indicate the amount of 
change. The flaws with this method are, 1) it does not capture non-critical systems that 
are being altered during the availability, 2) a ratio is intended to compare two similar 
values and in this model the two values are not similar and 3) the result is in a form of a 
percentage but it is unclear what it is a percentage represents. The solution to the first 
flaw is to calculate the significance to non-critical systems which will be discussed in the 
next section. The last flaws can be resolved by not doing the calculation and only using 
the sum of the warfare area significance scores. 
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5. Calculate Non-Critical System Significance 

The purpose of this function is to account for systems that are not critical to a 
major warfare area. All combat/weapon systems are important to the overall mission of 
the ship but some will not directly support primary warfare areas. Changes to these 
systems must be accounted for to determine if the modernization availability will have 
significant changes. The reason for accounting for non-critical systems is because any 
change will impact the crew’s ability to operate the equipment. For example, a hardware 
change to the air traffic control system may not impact air defense or surface warfare, but 
it does impact the ship’s ability to recover aircraft. These non-critical systems must be 
included in the overall assessment of significance. The non-critical system significance is 
calculated in the same way the warfare area significance is calculated; how it is 
implemented will be discussed in Chapter IV. 

B. STAGE 2 

The risk assessment tool process addresses risk to a ship’s readiness after an 
overhaul/modemization availability. The current process used by NSWC PHD is still 
under development and has not been implemented to assess the risk for any ship 
conducting a modernization availability. For this reason, the best approach is to 
determine a way to assess the risk associated with a major overhaul. After a practical 
method is developed to determine the risk, the next step is to review the proposed process 
discussed in Chapter II. 

1. Risk Assessment Methodology 

The difficulty with risk assessment is trying to predict the future based on current 
knowledge. The first step for assessing risks requires understanding all the possible 
outcomes, determining what can cause those outcomes, and the probability of each cause 
occurring. The next step is to apply a severity to each outcome; the greater the severity, 
the greater the risk. Risk is defined as a measure of the probability and severity of 
adverse effects (Haimes 2009, 4). There are many methods that can be used to assess risk, 
but all methods essentially start with the following three questions: 



• What is the likelihood that it would go wrong? 

• What are the consequences? (Haimes 2009, 24) 

The methodology for this research will begin with these three questions. To begin 
the assessment it is important to understand the goal of CSSQT. Figure 14 shows that the 
goal of CSSQT is to ensure a safe and effective combat system. 


Goal : Safe and EfEecCive CombaC Systeir. 


Act'ompl i shiid by: PMS/MRC and Liv(i/Siinulared Exerciues 


Maintenance 

Readiness 


Operational 

Headinesa 


Safety 





• PMS/MRC 


• Live/Siraulated 


• Ordnance Handling 

• System Documentecion 


Operational 


• Ordnance Movement 

Adequacy 


Exercises 


• Loading Procedures 

• Logistics Support 


• PEO Requiiemer.ts 



Adequacy 


• TYCOM 



• Maintenance Adequacy 


Requirements 



• Operational Adequacy 






Figure 13. CSSQT Goals, Objectives, and Requirements (From NAVSEA 2006) 


The three areas that ensure a safe and effective combat system are maintenance 
readiness, operational readiness and safety. Figure 13 provides categories for answers to 
the first question of “What can go wrong?” They include: the ship may not be able to 
accomplish maintenance actions, the ship may not be operationally ready, and the crew 
may not be able to safely handle ammunition. The third question focuses on the 
consequences from the answers to what can go wrong. If a ship is not operationally 
ready, then the training cycle may take longer or require more man hours and could lead 
to schedule changes or delayed deployments. If the crew is unable to maintain the new 
equipment, then the consequences could be increased technical assistance, and longer 
equipment downtime. This risk assessment will not focus on the consequences but only, 
address the likelihood of what can go wrong. 

Answering the second question about the likelihood of a risk occurring is what 
this risk assessment is going to accomplish. CSSQT ensures the ship has a safe and 
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effective combat system after significant overhaul/modemization, but what is the 
likelihood of a ship not being safe and effective if a CSSQT is not conducted? The 
significance assessment may result in an overhaul/modernization being classified as 
significant but, the assessment is based on the type of changes and how the changes 
impact the warfare areas. The risk assessment addresses how the entire availability, not 
only the modifications, affect the three areas reviewed by CSSQT. If the maintenance 
availability does not impact the maintenance readiness, operational readiness, or safety of 
the ship, than it may not be necessary to conduct a CSSQT even if the availability is 
deemed significant. Cause-Effect diagraming will be used to determine how the 
availability, as a whole, impact the three focuses of CSSQT. 

During the availability, what is being added or changed, how long is the 
availability, what is the time lapse since training cycle, or whether there is crew turnover 
can have great impact on how well the ship’s force can maintain and operate its new 
combat system. A cause and effect diagram can be used to show which areas are at risk. 
Figure 15 depicts how a safe and effective combat system is achieved by conducting a 
CSSQT. 


I Ensure Maintenance Readiness! 


CSSQT Cause and Effect Diagrarr 


Software Change 


HardwareChange - 


- Maintenance Adequacy 


I Ensure Operational Readinessj 


' Training 

\ DTE Impact Percentage of Systems Changed 
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Length of Availability 
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- Number ofWeapon Systems Changed 


I Ensure Safety! 


Figure 14. Cause-Effect Diagram 
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Cause and Effect diagrams are usually used to find the root cause of a problem, 
but in this case the goal is to find what influences a positive outcome. The diagram can 
also be used to determine the risks to the causes. Each cause can be broken down into 
secondary causes and these causes can be used to determine what drives each primary 
cause. Eor example, CSSQT ensures operational readiness by performing live and/or 
simulated operational exercises. If operational exercises are not performed by the crew 
then operational readiness cannot be ensured; therefore, the ship has a risk of not having a 
safe and effective combat system. The secondary cause can be broken down further (if 
required) to address potential risk factors that can be used to assess the risk to each 
primary cause. Each arrow on the diagram represents a risk factor. Each risk factor needs 
to be investigated to determine a way to quantify the risk. 

The three area of focus are maintenance readiness, operational readiness, and 
safety; these are the risk areas for overall ship readiness. Measuring the risk to the ship’s 
readiness will be a function of the risk to the three risk areas. The goal is to quantify the 
risk in a way the represents the likelihood of risk; that is, the higher the value the more 
likely the ship will not be operationally ready. The next sections will break down each 
risk area and discuss measuring the risk. 

a. Risk to Maintenance Readiness 

Maintenance readiness is the ship’s ability to maintain the combat/weapon 
systems onboard. Prior to entering a major availability, the crew has the right people 
trained to maintain the equipment and will have accumulated months of experience 
maintaining it. After the availability, the maintenance readiness may be decreased 
because a system that has been upgraded, replaced or installed may affect the hardware, 
software, planned maintenance system, technical manuals, supply parts, certifications. 
Operating Sequencing System (OSS), detect-to-engage path, training, and test 
equipment. These are the risk factors taken from the cause and effect diagram. Not all of 
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these factors will be affected, so the assessment of risk to maintenance readiness will 
determine what is being changed and quantify the changes. 

The main source of data for impact to maintenance readiness is a SCD. As 
discussed previously, SCDs contain information about administrative functions and the 
impact on maintenance readiness. A SCD consists of 26 sections and the following 
sections are used to determine the risk to maintenance readiness: 

• Section 4a and 4b: Scope and Category 

• Section 11: Executive Summary 

• Section 12: Description of Change 

• Section 13: Impact if not Accomplished 

• Section 14: Requirements and Justification of Change 

• Section 15: Distributive Systems/Other Impacts 

• Section 16: Human Systems Integration Impacts 

• Section 17: Integrated Logistics Support (ILS) Impact 

• Section 23: Certifications/Qualifications 

The scope and category will indicate if it is a hardware change, software 
change or both. Sections 11, 12, 13, and 14 can all be used to understand the nature of the 
change, such as if a new system is replacing an old system. Section 15 describes how the 
alteration impacts other systems on the ship. Section 16 will indicate how training is 
impacted. Section 17 lists all ILS systems that are impacted and Section 23 lists the 
certifications that will be required after the installation. SCDs, in most cases, do not 
describe the impact a change will have on a specific maintenance area. 

Due to the size of an SCD only one section is presented here to 
demonstrate how the information is presented. Ligure 16 is an example of what is found 
in section 17 of a SCD. The only information that can be inferred from this section is the 
change will affect technical manuals and provisioning. 
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Steam Plant Manual 

E 

Test Equipment 

E 

COTS/NDI 

E 

Facilities 

Figure 15. 

SCD Section 17: Integrated Logistics Support (ILS) Impact 


It is not known if technical manuals will be added, removed or updated or how the 
supply systems (provisioning) will be impacted. If this information was available it may 
be possible to use an assessment approach that uses the number of changes being made to 
each area. For example, if the system before the change required 45 spare parts onboard 
and after the change the system will require 60 spare parts than the 33% increase in the 
number of parts could be used to indicate a certain likelihood that maintenance readiness 
will be impacted. The reason maintenance readiness could be impacted is because of the 
possibility the supply system does not get updated with the correct parts. If the ship is not 
stocked with the correct parts than a delay in repair could occur causing longer downtime 
of the system. CSSQT verifies that the ship has the correct parts onboard for each system 
therefore reducing the likelihood of this situation occurring. The assessment, if this 
information was present, could set thresholds for the amount of change being made to an 
area and make a better determination of risk if the threshold is exceeded. 

Information on how each area is being affected is not available therefore, a binary 
system is the best approach to evaluate the changes being made to the system; if a box is 
checked then the category receives a one, if not checked it receives a 0. This method can 
be used on each system being impacted by the availability. There are ten categories that 
relate to maintenance readiness, so if a specific system scores a 10 then all areas are 
being affected by the change. The average score of the all the systems will yield a value 
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that is indicative of how much the availability is impacting the maintenance readiness. 
The higher the score, the more likely there will be a risk to maintenance readiness post¬ 
availability. 


b. Risk to Operational Readiness 

Operational readiness is the adequacy at which the ship’s force can 
operate its combat system and the ability of the equipment to operate as designed. The 
secondary indicator of operational readiness is performance of live and/or simulated 
operational exercises. When a ship conducts a CSSQT it will perform multiple simulated 
warfare specific scenarios that stress the combat/weapon systems and the ship’s crew. 
The final exercises will usually involve live firing of missiles and guns against unmanned 
drones and other targets. Most ships will only get to conduct live firing during CSSQT, 
but all ships conduct simulated exercises during the training cycle and during 
deployments. Determining the risk to operational readiness involves these two aspects of 
a ship’s life: Time lapse since last CSSQT and length of availability. The latter is under 
the assumption that a majority of ships return from deployment and enter the 
maintenance phase. A third aspect that can be used to determine operational readiness 
risk is the percentage of systems modified during an availability. The more systems that 
are changed, in most cases, the less familiar the crew will be with operating them. The 
ability to use the systems as designed in a stressing situation decreases because of the 
novelty of the revised system. The likelihood of risk to operational readiness will be a 
function of time lapse, length of availability and percentage of systems changed. The 
likelihood will range from 0 to 10 where 10 would indicate the greatest likelihood. 

c. Risk to Safety 

The area of safety CSSQT addresses is ordnance safety. Ordnance 
handling, movement, and loading is a high risk evolution and if a ship has not had 
training on the proper procedures for an extended period of time, the risk is increased. 
What can go wrong is the mishandling of ordnance or improper loading procedures 
which can lead to mishaps that can cause injury or even death. The likelihood of the ship 
not being safe is a function of the weapons systems being altered during the availability. 
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If an availability will affect all weapon systems onboard than the likelihood of risk is the 
greatest. In this case the ship will receive a score of 10. If no weapon systems are being 
upgraded or added then the risk is the lowest yielding a score of 0. When there is one or 
more weapon system is being modified, but not all, then the score will be a 5. 

2. Review of Current Risk Assessment Process 

The risk assessment methodology shows the functions required for the assessment 
are determining the risk areas, the risk factors for each area, and a way to quantify the 
risk factors. Another function, that was not discussed, is the ability to repeat the 
assessment in a way that will ensure consistent results. This means that the data used to 
conduct the assessment must lack any sort of subjective assessment. NSWC PHD has 
developed a process to assess risk and it was described in Chapter II. This process 
identified four risk areas and 18 risk factors. It also quantified the risk factors so they 
could be used to formulate a value for likelihood. The issue with the proposed process is 
that it relies on factors that are subjective. 

There are five risk factors in the proposed process that solicit an evaluation of a 
SME. The five factors are: 

• Ship’s force ability to perform the mission 

• Known system of incidents of risks 

• Known safety incidents of risks 

• Project Officer/Project Engineer evaluation of risk 

• Baseline Manager/Equivalent evaluation of risk 

The evaluation by SMEs can differ greatly from ship to ship and it is important all 
subjective assessments are removed when conducting a quantitative assessment. It also 
introduced a level of complexity because the execution of the assessment relies on the 
input of outside entities. There are other risk factors that rely on the input from other 
agencies to complete the assessment and those are: 

• Training program planned or developed/delivered(AIT) 

• Training program planned or developed/delivered(EBTS) 

• Planned ATG Training 
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• Manpower requirements adequately reflected 
The problem with using the inputs from outside sources is potential delays in gathering 
information. In any given fiscal year there will a large number of ships conducting an 
availability and it is the advantage to the assessors to rely on one or two sources for data. 
The current process could take an assessor weeks to complete while the methodology 
described in the previous section is able to be completed in days. 

The other issue with the risk assessment process proposed by NSWC PHD is that 
it does not follow the goals of the NAVSEAINST 9093.1C. The instruction specifically 
outlines what CSSQT accomplishes and the risk assessment should coincide with the 
objectives of the instruction. There are only three areas that CSSQT addresses but the 
proposed process describes four areas. Each area in the instruction had specific factors 
that are reviewed by the CSSQT but the proposed process has multiple factors that apply 
to more than one of the four areas. The risk assessment should be precise and lack any 
ambiguity; however, the proposed risk assessment process is full of ambiguity and does 
not provide a precise measure of the likelihood that the ship’s readiness is at risk. 

C. STAGE 3 

The purpose of this stage is to develop a methodology for presenting the results 
from the significance assessment methodology and risk assessment methodology. The 
current process used by NSWC PHD uses a separate spreadsheet to assess the 
significance and another one for the risk assessment. The only interface between the two 
processes is that the result of the significance assessment is used in the risk assessment as 
a risk factor. NSWC PHD has proposed that the results of the assessment will be reported 
on a single page that is divided into four quadrants (commonly referred to as a quad- 
slide) where one quadrant gives information about the ship, another one for 
administrative data, and the last two show the results of the assessments. The issue with 
this approach is that there is no coupling of results of the assessments. 

The end result of the two assessments should result in an understanding of the 
overall impact to the ship. Overall impact is a function of the how significant the changes 
are and the likelihood that the changes will affect the ship’s readiness. The presented 


33 



methodologies put significance and likelihood on the same sized scale so it makes it 
possible plot the two areas in a matrix. The reason for plotting the two areas is to give a 
visual representation of the results. When each ship is plotted on the same matrix, 
decision-makers will have the ability to determine which ship will be impacted the 
greatest and also know which ships can proceed without a CSSQT. The approach for 
presenting the results will be to use a summary spreadsheet that captures each ship 
conducting an availability in a given fiscal year, information about each ship’s 
availability and history, the results of the significance assessment, the results of the 
likelihood assessment and a matrix that plots each ship. 

This chapter developed methodologies for determining if the changes being made 
to a ship during a modernization are significant, if the changes being made impact the 
readiness of the ship, and a way to present the results. It also reviewed the current tools 
being used by NSWC PHD to determine shortfalls and issues. The review found that 
there are issues with the current tools that need to be addressed. The next chapter will use 
the methodologies to develop a tool that will determine significance and likelihood and 
present the results. 
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IV. RESULTS AND ANALYSIS 


The previous chapter discussed issues with the current process for determining the 
impact to a ship’s readiness after a major availability and also developed methodologies 
to implement a better process. The results of implementing the methodologies will be 
described in this chapter. The first step designs a spreadsheet that presents the 
information about each ship being assessed, the results of the significance assessment, 
and the results of the risk assessment. The second step is to conduct the significance 
assessment and risk to maintenance readiness assessment. The final step plots the 
significance and risk likelihood for each ship on a matrix. The end product will be a 
single spreadsheet that presents the results of the significance assessment, risk assessment 
and matrix. 

The analysis will use data about ships that are scheduled to conduct a major 
availability in fiscal year 2015. The ships used for the assessment are: 

• USS Nimitz (CVN 68) 

• USS George HW Bush (CVN 77) 

• USS Harry S Truman (CVN 75) 

• USS Ronald Reagan (CVN 76) 

• USS Germantown (LSD 42) 

• USS Gunston Hall (LSD 44) 

• USS Ashland (LSD 48) 

• USS Carter Hall (LSD 50) 

• USS Pearl Harbor (LSD 52) 

• USS Makin Island (LHD 8) 

All information needed to conduct the assessment is available in three locations: 
WISE Database, Navy Data Environment (NDE), and NSWC PHD ship information 
database. The WISE database contains information about the availability and the systems 
onboard the ship. SCDs are located in the NDE and the ship information database has the 
results of the previously conducted CSSQT. 
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A. CSSQT DETERMINATION DASHBOARD 

The CSSQT Determination Dashboard is a central location that presents the 
information gathered during the assessment. The dashboard uses information about the 
availability, the results of the assessments and does calculations to determine the 
significance of changes and risk likelihood. Figure 17 shows what a completed dashboard 
looks like. 
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Figure 16. Completed CSSQT Determination Assessment 


Figure 16 is used to show the design of the dashboard; but, each section will 
discussed in depth for better understanding. 

I. Availability Data 

Figure 18 shows the first section in the Determination Dashboard. The 
information in this section is found in the WISE database. It lists the ship’s hull number, 
the type availability, and the start and end date of the availability. Each ship is 
hyperlinked to the ship specific worksheet that contains the significance assessment and 
risk to maintenance assessment. 
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Eigure 17. Availability Data Section of Dashboard 
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2 . 


Significance Assessment 


Figure 19 shows the areas that make up the significance assessment section of the 
dashboard. AD Impact Score, SUW Impact Score and the Non-Critical Weight are 
extracted from the significance assessment worksheet for each ship. The significance 
score is the average of the AD Impact Score and SUW Impact Score plus the Non- 
Critical Weight. 
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Figure 18. Significance Assessment Section of Dashboard 

3. Risk Assessment 

Figure 20 shows the areas that make up the risk assessment section of the 
dashboard 


Risk Assessment 


Avail 


Total 

% 


Operational 

Maintenance 



Length 

Total 

Systems 

Systems 

Last 

Delta Readiness 

Readiness 

Safety 

Risk 

(Months) 

Systems 

Changed 

Changed 

CSSQT 

(Years) Score 

Score 

Score 

Likelihood 


Figure 19. Risk Assessment Section of Dashboard 


Availability length is the number of months of the availability. Total Systems is a 
count of the number of combat/weapon systems onboard the ship. Total Systems 
Changed is number of systems that are being changed or added during the availability. 
Both inputs are calculated from the significance assessment and the percentage of 
systems changed is the total systems changed divided by the total systems. Last CSSQT 
is the previous year the ship conducted a CSSQT. The number of years that have elapsed 
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since the last CSSQT and the end of the availability is the Delta. The next four areas of 
the risk assessment section are calculated from the previous sections and also use input 
from a separate worksheet. 

Operational Readiness score is a function of Availability Length, Percentage of 
Systems Changed, and Delta. The function is designed to give more weight to the time 
elapse since the last CSSQT. This is accomplished by weighting the sum of the Delta and 
length of availability. If the Delta is greater than 13 years the Operational Readiness 
score will be dependent on the percentage of systems changed. So, if a large percentage 
of systems are being affected and the delta between CSSQTs is greater than 13 years, the 
likelihood of risk to operational readiness is greater. When the elapse between CSSQTs is 
less than 13 years the operational readiness risk likelihood is less. The formula used to 
determine the Operational Readiness score in an Excel spreadsheet is 
MIN(10,LengthOfAvailabilty*0.25+Delta*0.75)*%SystemsChanged. The formula is 
designed to ensure the score of Operational Readiness is less than or equal to 10. The 
next areas are the risk to Maintenance Readiness and Safety. 

The risk to Maintenance Readiness Score comes from an assessment that is 
performed on the Significance Assessment worksheet. The details of the assessment 
procedure will be discussed in the next section. The Safety Score will be a 0, 5, or 10 
based on the number of weapon systems being altered during the availability. This score 
is determined by the user and can be found by looking at the Significance Assessment to 
find which weapon systems are being modified. The Safety Score can also be a 10 if the 
Significance Assessment shows that a new weapon system is being installed on the ship 
that will require new ammunition handling and loading procedures. The final value in the 
Risk Assessment section is the overall Risk Likelihood. It is the average of the 
Operational Readiness, Maintenance Readiness and Safety scores. 
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B. SIGNIFICANCE ASSESSMENT AND RISK TO MAINTENANCE 

READINESS ASSESSMENT 

Four values in the Determination Dashboard are determined on a separate 
worksheet within the spreadsheet that are specific to each ship being assessed. The 
significance assessment will calculate how significant the changes are to each warfare 
area and non-critical systems. The maintenance readiness assessment will calculate the 
likelihood of risk to maintenance. 

I. Significance Assessment 

NSWC PHD called this assessment the CSSQT Decision Tool Process. The 
review found areas of the process that needed improvement and the name is the first issue 
that needs to be addressed. The process is not making a decision; rather, it is assessing if 
the changes being made to the combat/weapon systems are significant. It is a minor 
change, but gives more clarity to the purpose of the process. The second issue found was 
the accuracy of the systems installed onboard each ship. There were numerous ships that 
had the wrong systems listed and/or systems were missing. This is easily addressed by 
double checking that systems input on the worksheet are the same systems listed on the 
WSID. The third issue was the mission area criticality and modernization type categories 
and subsequent weights which will be discussed next. 

The review of the process found that systems that were not essential to a specific 
warfare area were not assessed even if the systems were being altered. All 
combat/weapon systems having any sort of modification need to be included in the 
assessment regardless of the importance to a warfare area. This issue is addressed by 
giving weight to non-critical systems. The modernization type categories did not have a 
category for systems not being changed which led to a spreadsheet that contained a 
number of blank cells and cells that contained FALSE as an entry. When a spreadsheet 
analysis has blank cells and FALSE cells, it looks incomplete. This is addressed by 
adding a sixth category to the modernization types. The new categories and weights used 
in the Significance Assessment are shown in Eigure 21. 
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Figure 20. Updated Categories and Weights 


The final issue the review found with the process to determine if the changes are 
significant is the calculation of the significance value. As discussed in Chapter III, each 
system received a score by multiplying the mission criticality weight by the 
modernization type weight. The sum of all the systems scores yielded a total score for the 
warfare area. The process then would sum the warfare area criticality weights and divide 
the total score for the warfare area by this sum. This ratio was the value used to show 
how the changes impact each warfare area. The average of the warfare area scores would 
be the overall impact to the ship. 

The methodology in Chapter III determines a better way to calculate if the 
changes are significant is to use only the sum of the system scores since this value takes 
into account both criticality and the type of modernization. Another problem with the 
calculation was it did not account for non-critical systems. The score for non-critical 
systems is calculated in the same manner as the score for a warfare area. The score for 
each warfare area and non-critical systems is used in the Determination Dashboard to 
calculate the overall significance of changes. 
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The assessment worksheet is large and a completed assessment is in Appendix D. 
Figure 22 shows a portion of the completed assessment. 
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Figure 21. Portion of Completed Significance Assessment 


The ship used for the assessment in Figure 21 has 29 total combat/weapon 
systems onboard. The AD Impact Score, SUW Impact Score, and Non-Critical Weight 
columns are summed and this sum is the value used in the Determination Dashboard. 

2. Risk to Maintenance Readiness 

The methodology described in Chapter III determined the best method for 
assessing the risk to maintenance readiness is use a binary system. The first step is to 
create the assessment workspace. The logical place to conduct the assessment is on the 
same worksheet the Significance Assessment is conducted because the risk assessment is 
analyzing systems that are being modified during the availability. Each system that is 
being altered is copied from the list of systems in the Significance Assessment and listed 
in an area below the Significance Assessment. The second step is to label columns with 
the risk factors from the methodology. The risk factors are: 

• Hardware Change 

• Software Change 
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• PMS Impacted 

• Technical Manuals Impacted 

• Part Supply Impacted 

• Training Impacted 

• Certification Required 

• Test Equipment Impacted 

• Operating Sequencing System Impacted 

• Detect-to-Engage Path Impacted 

The third step is to collect all the SCDs applicable to the ship from the NDE. In 
most cases, when there is a software change and a hardware change on the same system, 
there will be two separate SCDs. The SCD sections from the methodology are reviewed 
and if the SCD indicates the change will impact a risk factor, the fourth step is to enter a 
1 in the column. If it is not being affected, a zero is entered. The final step is to sum the 
values for each system to get a risk score for each system and then sum the risk scores to 
get an overall risk to maintenance readiness score which is used in the Determination 
Dashboard. Eigure 23 shows a completed assessment (broken up for display purposes). 
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Impacted System 

Hardvt/aro Change 

Software 

Change 

PMS Impacted 

Tech Manuals 
Impacted 

Part Supply Impacted 

3D Volume Search Radar (SPS-48) 

1 

1 

1 

1 

1 

High Fidel ity Radar RADAR (5PQ-9B) 

0 

1 

0 

1 

0 

Navigation Radar (SPS-73) 

1 

1 

1 

1 

1 

Surface Search Radar {SPS-67) 

1 

0 

1 

1 

1 

MK38 MGS 

1 

1 

1 

1 

1 

Critical Navigation Distribution (NCDS) 

0 

1 

0 

1 

0 

Command and Control (SSDS) 

1 

1 

1 

1 

1 

BFH 

0 

1 

0 

1 

0 

TSSS 

1 

1 

1 

1 

1 

Electronic Signal Detection (SLQ-32) 

1 

1 

1 

1 

1 

Self Defense (RAM) 

1 

1 

1 

1 

1 

Point Defense (CIWS) 

1 

1 

1 

1 

1 

CDLMS 

0 

1 

0 

1 

0 


Training 

Impacted 

Certifications 

Required 

Test Equipment 
Impacted 

Operating 

Sequence 

Systems 

Impacted 

DTE 

Impacted 

Sum 


1 

1 

0 

1 

1 

9 

1 

1 

0 

0 

1 

5 

1 

1 

0 

0 

0 

7 

0 

1 

0 

0 

1 

6 

1 

1 

0 

1 

1 

9 

0 

1 

0 

0 

0 

3 

1 

1 

0 

0 

1 

8 

1 

1 

0 

0 

0 

4 

1 

1 

0 

0 

0 

7 

1 

1 

0 

1 

1 

9 

1 

1 

0 

1 

1 

9 

1 

1 

0 

0 

1 

8 

1 

1 

0 

0 

0 

4 



6.77 


Figure 22. Completed Risk to Maintenance Readiness Assessment 


C. CSSQT DETERMINATION MATRIX 

The final piece of the CSSQT Determination Dashboard is the Determination 
Matrix. When the all information is gathered and inputted for each ship, the final overall 
assessment for the fiscal year will look like Figure 24. 
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Availability Data 

Significance Assessment 

Ship 

Avail 

Avail Start 

Avail End 

AD 

Impact 

Score 

SUW 

Impact 

Score 

Non-Crit 

Weight 

Significance 

Score 

CVN68 

PIA3 

May-14 

Feb-15 

5.13 

4.13 

0.31 

4.94 

CVN 77 

PIAl 

Apr-15 

Oct-15 

5.63 

3.88 

0.69 

5.44 


SRA 



1.00 

1.00 

0.00 

1.00 




■m 

5.00 

4.75 

0.69 

5.56 




Nov-14 

2.50 

3.13 

0.00 

2.81 

LSD 50 

EDPMA 

Jan-14 

Jan-15 

6.25 

6.00 

0.75 

6.88 

LSD 52 

EDPMA 

Feb-14 

Feb-15 

6.25 

5.13 

0.63 

6.31 

CVN 75 


Sep-14 

Mar-15 


3.00 

1.25 

5.50 

CVN 76 


Ort-14 

Apr-15 

5.00 

4.00 

2.00 

6.50 

LHD8 

PMAl-2 

Feb-15 

Aug-15 

3.75 

3.50 

0.13 

3.75 



Risk Assessment 

Avail 

Length 

(Months) 

Total 

Systems 

Total 

Systems 

Changed 

% 

Systems 

Changed 

Last 

CSSQT 

Delta 

(Years) 

Operational 

Readiness 

Score 

Maintenance 

Readiness 

Score 

Safety 

Score 

Risk 

Likelihood 

9.00 

29 

13 

45% 

2007 

8 

3.70 

6.77 

10 

6.82 

6.00 

31 

13 



6 

2.52 

7.08 

5 

4.86 

0.25 

18 

2 



29 

1.11 

5.5 

5 

3.87 

0.33 

20 

14 



26 

7.00 

6.57 

10 

7.86 

0.25 

19 

7 



22 

3.68 

5.71 

5 

4.80 

1.00 

20 

15 


1995 

20 

7.50 

6.80 

10 

8.10 

1.00 

20 

14 


1998 

17 

7.00 

6.80 

10 

7.93 


31 

16 

52% 

2007 

8 

3.16 

7.00 

10 

6.72 


32 

16 

50% 

2008 

7 

2.69 


10 

6.48 

6.00 

29 

8 

28% 

2009 

6 

1.66 

7.5 

10 

6.39 


Figure 23. Completed FY15 CSSQT Determination Dashboard 


The Signifieance Score and Risk Likelihood (highlighted with dark boxes) are the 
values plotted on the Determination Matrix chart shown in Figure 25. 
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FY15 CSSQT Determination Matrix 
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Figure 24. FY15 CSSQT Determination Matrix 


The matrix is divided into four quadrants and each quadrant is labeled with a 
recommendation. If a ship is plotted in quadrant one a CSSQT is recommended because 
there are significant changes being made to the ship, but the likelihood that ship’s 
readiness will be at risk is low. A ship in quadrant two has significant changes being 
made and the likelihood that the changes will impact ship’s readiness is high; therefore, it 
is recommended that a CSSQT is required for that ship. In quadrant three the changes are 
not significant but the likelihood that the changes will impact ship readiness is high, so 
the ship is recommended to conduct a CSSQT. A ship in quadrant four does not have 
significant amount of changes and the changes have less likelihood to impact ship’s 
readiness and it is recommended that a ship in the fourth quadrant does not conduct a 
CSSQT. Ships in quadrants one and four are recommended for CSSQT which means if 
funding and resources are available then these ships should be designated to perform a 
CSSQT. Ships in quadrant two would benefit the most from a CSSQT and it those ships 
should be the first to be allocated funding and resources. 
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This chapter used the methodology and the results of the review of eurrent work 
diseussed in Chapter III to develop a better way to assess the impaet a maintenanee 
availability has on a ship’s readiness. The CSSQT Determination Dashboard was 
designed to put all the results of the assessment in a eentral loeation. The CSSQT 
Deeision Tool proeess used by NSWC PHD was redesigned and the name was ehanged 
to Signifieance Assessment for elarity about the purpose of the tool. The risk assessment 
process was eompletely ehanged to align with NAVSEAINST 9093.1C and to remove 
subjeetive assessments. Finally, the Determination Matrix was introduced to graphically 
show the impaet to ship’s readiness. The next and final ehapter will diseuss eonelusions 
and reeommendations for further researeh. 
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V. CONCLUSION AND RECOMMENDATIONS 


The purpose of this thesis is to develop a process to recommend ships for CSSQT 
by determining the impact an overhaul/modernization has on a ship’s readiness. This was 
accomplished by analyzing a process used by NSWC PHD to recommend ships for 
CSSQT and determine if the process can be improved. The ultimate goal of the process is 
to determine the impact on a ship’s readiness based on how significant the changes are 
and the likelihood the changes will impact readiness. The results of the process aid 
decision-makers in determining if a CSSQT is required for a ship post-availability. The 
thesis presented background information about CSSQT and the current process used by 
NSWC PHD. A majority of the work focused on a creating a methodology for reviewing 
the current process and an approach to correct issues discovered during the review. 
Finally, the work resulted in an assessment process that quantifies the impact a major 
overhaul/modemization has on a ship’s readiness in a valid and objective way. 

A. SUMMARY OF FINDINGS 

The review of the current tools used by NSWC PHD was intended to investigate 
the processes used for flaws and issues. The analysis a determined that the method for 
determining significance was flawed for the following reasons: 

• The name of tool does not accurately reflect the purpose of the tool 

• There is an inaccurate accountability of all combat/weapon systems 

• The significance calculation result is ambiguous 

• The significance calculation does not account for non-critical systems 

• A completed assessment can result in a spreadsheet that contains empty 
cells and cells that contain FALSE as an entry 

The risk assessment review found the following issues with the proposed process: 

• The risk areas do not align with the objectives outlined in NAVSEAINST 
9093.1C 

• Risk factors applied to more than one risk area which made the assessment 
complicated 

• A large number of risk factors required subjective input from outside 
sources 
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• The output of the assessment was ambiguous 

The byproducts of the review were a better understanding of the purpose of the 
assessment and a new methodology for conducting the assessment. The methodology 
determined the key to the assessment is the result has to be on a measurable scale. That is 
how ambiguity is removed from the output. By setting a scale and putting the result on 
that scale, there is more clarity in what the result means. The methodology used a 0 to 10 
scale for both significance and risk likelihood. The higher the result of the assessment the 
more significant the changes are and the greater likelihood the changes will impact ship’s 
readiness. Using the same scale for each assessment also allows the results to be plotted 
on a graph for better overall understanding and comparison with other ship being 
assessed. 

The approach developed in this thesis is not perfect, but it does present a way to 
assess the impact a major overhaul/modernization will have on a ship. The assessment is 
completely quantitative and will yield dependable results when repeated for multiple 
ships. It also gives decision-makers a level of detail that does not overwhelm but enough 
to make a sound decision about moving forward or not in designating a ship for CSSQT. 

B. RECOMMENDATIONS FOR FURTHER STUDY 

Assessments that include risk are better when there is historical data available. 
The next logical step in for assessing the impact a maintenance availability is to collect 
data about the ship’s performance post-availability. The data needs to include ships that 
conduct a CSSQT and those that do not to determine if there is a correlation between 
ship’s readiness and the conduction of a CSSQT. There could be other correlations, such 
as length of availability and ship’s readiness or number of systems changed and ship’s 
readiness. The result would be thresholds that each ship could be compared to. For 
example, a ship that is conducting a 12 month availability and is having 75% of its 
combat/weapon systems altered should conduct a CSSQT because historical data shows 
that ships having these characteristics, and did not conduct a CSSQT, are more likely to 
be delayed in the training cycle. The research would require determining what data to 
collect and which metrics should be analyzed. 
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This research was only conducted on three classes of surface ships but CSSQT is 
conducted on all surface ships. The tool needs to be expanded to assess other classes of 
ships so there is a common process for designating all surface ship for CSSQT. This 
could be accomplished by enhancing the CSSQT Determination Process by using 
computer software that allows users to easily complete an assessment. Using a software 
program could allow users to select the ship class and it automatically generates an 
interface to input required information. The use of VBA is also a potential way to make 
the process simpler. 

The final area that requires further research is the periodicity of CSSQT. CSSQT 
is a valuable evolutions that benefits both the crew and the ship. For some ships, a 
CSSQT is only conducted once in the entire life of the ship. An analysis of the 
importance of CSSQT should be conducted to determine if a CSSQT should be 
conducted on a more routine schedule. 
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APPENDIX A 
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NSWC PHD CSSQT DETERMINATION PROCESS 
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APPENDIX B 
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APPENDIX C 
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CDLMS Like to have 0.25 Like to have 0.25 improve Performance 0.5 0.125 0.125 

11.25 8.50 2.13 0.38 
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Ship: _ USS Nimitz (CVN-68) _ Avail: FY14 DPIA3 

Significance Assessment 
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Total Systems 29 

Systems Changed 13 

% Changed 45% 
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